DOCUMENT RESUME 



ED 269 253 

AUTHOR 
VITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



SE 046 571 

Raizen, Senta A. 

Report on the Work of the Committee on Indicators of 
Precollege Education in Science and Mathematics. 
[86] 

25p.; Paper presented at the Annual Meeting of the 
American Educational Research Association (70th, San 
Francisco, CA, April 16-20, 1986). 
keports - Research/Technical (143) — 
Speeches/Conference Papers (150) 

MFOl/PCOl Plus Postage. 

*Educational Improvement; *Educational Quality; 
Elementary Secondary Education; Evaluation Criteria; 
*Mathematics Education; *Performance Factors; 
*Science Education: *Scientific Literacy 
*Indicators 



ABSTRACT 

This commitee of the National Research Council was 
established to develop improved indicators of mathematics and science 
education. In its first report, the committee used a model of 
education consisting of inputs, processes, and outcomes. Indicators 
recommended in these areas include teacher quantity and quality, 
quality of the curriculum content, instructional time and course 
enrollment in science and mathematics, and student achievement. Data 
and information currently available on these indicators are 
summarized and their auaquacy assessed, exclusions on how to improve 
the database are provided. The committee's current work is also 
summarized. Areas currently under consideration for development of 
indicators include resource investments in mathematics and science 
education, assessing teaching effectiveness, further consideraton of 
curriculum quality indicators, improving the assessment of learning, 
assessing attitudes and motivation, and indicators of icience 
literacy in the general population. (Author) 
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This committee of the National Research Council was established to 
develop improved indicators of mathematics and science education. In its 
first report, the committee used a model of education consisting of 
inputs, processes, and outcomes. Indicators recommended in these areas 
include teacher quantity and quality, quality of r'<e curriculum content, 
instructional time and course enrollment in science and mathematics, and 
student achievement. Data and information currently available on these 
indicators are summarized and their adequacy assessed. Conclusions on how 
to improve the data base are provided. The committee's current work is 
also summarized. Areas currently under consideration for development of 
indicators include resource investments in mathematics and science 
education, assessing teaching effectiveness, further consideration o^ 
curriculum quality indicators, improving the assessment of learning, 
assessing attitudes and motivation, and indicators of science literacy in 
the geueral population. 
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REPORT ON THE WORK OF THE 
CGMMirrSE ON INDICATORS OF PRECOLLEGE 
EDUCATION IN SCIENCE AND MATHEMATICS 

Presentation for the AERA 1986 Annual Meeting 
Senta A. Ralzen 



This presentation summarizes the work of a committee of the National 
Research Council which has been established to develop Improved indica- 
tors cf the condition of science ^nd mathematics education In the 
natlor/s schools. The Impetus for the work came from the convocation 
held by the National Academy of Sciences In spring 1982 on mathematics 
and science education (National Academy of Sciences, 1982) and the var- 
ious repor ts on the condition of education that appeared In the fall of 
1982 axA s^.rlng 1983 (see, for example, National Science Foundation, 
1983; National Commission on Excellence In Education, 1983; Twentieth 
Century Fund, 1983; Boyer, 1983). These reports all found serious 
Inadequacies In precollege education; a number of them suggested that 
most U.S. students leave high school without adequate preparation In 
science and mathematics, whether for the job market or for continuing 
their education. Ihe reports Identified such specific school deficien- 
cies aK teacher shortages. Inadequate curricula, and low ptandards of 
snudeat perf o^miance. Perusal of these reports led to widespread con- 
cerns about the state of schooling. However, there were also questions 
raised about the quality of the Information used to formulate the con- 
clusions and policy recommendations In the reports (see for example, 
Peterson, 1983; Stedman and Smith, 1983). This concern led to the 
creation of the NRC committee. The committee is charged with laying 
the foundation for the development of an adequate monitoring system for 
use at the national, state, and local levels, so that the condition of 
mathemathics and science education, could be tracked, particularly the 
effects of current: Improvement efforts. 

The committee was confronted with defining what an indicator of 
science and mathematics education might be, selecting— at least on a 
preliminary basis — particular schooling variables aa a basis for con- 
structing indicators, and reviewing the available data pertinent to the 
selected variables. A major problem, in fact, is the large amount of 
statistical data and research information available on education^ These 
data derive from diverse sources, address similar questions differently, 
and are collected and analyzed with v&rious degrees of rigor. Hence, 
they cannot readily be cumulated nor adapted for use as indicators. 
Moreover, despite the wealth of information, some pertinent issues are 
not addressed. The potential for confusion and misuse of data has been 
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vividly Illustrated In recent attempts to report on the nation *s 
schools. Therefore, the committee early In Its work decided to select 
a quite limited set of indicators and concentrate on assessing the 
quality of pertinent Information. Selecting a limited number of indi- 
cators, at least initially, is also important if one wishes to develop 
a monitoring system that is feasible and affordable at the federal , 
state, and local levels. (It is InLtructive to remember that econo- 
mists use nearly a hundred indicators to monitor the state of the 
nation's economy, even though only a few of these — gross national prod- 
uct, unemployment rate, cost of living. Interest rates — are gener^^lly 
reported by the mass media.) 

Selecting Indicators 

The committee chose a simple model of the education system for 
identifying areas of science and mathematics education to be monitored, 
consisting of educational inputs, schooling processes, and student out- 
comes. Since the primary goal of instruction in science and mathe- 
matics is student learning, the most explicit: student outcome is student 
achievement in these fields. Other outcomes such as choice of college 
majors or careers and later career paths are also important to society^ 
but it is more difficult to tie them directly to schooling variables. 
Another student outcome, student motivation and attitudes toward science 
and mathematics, was considered but not treated in the committee's pre- 
liminary work. Choosing student achievement as the outcome variable of 
greatest Interest determines to a considerable extect what schooling 
input and process variables need to be selected — those that have some 
causal relationship to student achievement. Based on research evidence 
as well as on educational practice and experience, time spent on subject 
matter was selected as a proxy for schooling processes that should be 
monitored. The most obvious schooling inputs linked to student achieve- 
ment in science and mathematics are the numbers (and qua],lty) of 
teachers responsible for these areas of Instruction and the content of 
the curriculum. 

For each of th«*se four ar^as — teacher quantity and quality, curric- 
ulum content, instructional tlttiv^ (and course enrollment in secondary 
school), and student achievement — the committee reviewed the data and 
information currently available, provided some findings on temporal 
trends and comparisons with other countries, judged the adequacy of the 
information available on the selected indicators, and made recommenda- 
tions for Improvement. (For details on the data sources and research 
reviewed, see Ralzen and Jones, 1985.) This summary of the committee's 
work will concentrate on the adequacy of the information available 
rather than on the state of science and mathematics education, although 
I would be glad to comment on the latter in the discussion period. 



Before summarizing the committee's conclusions and recommendations on 
the adequacy of Information, however, some general comments are In 
order: 

• Most of the pertinent data have been collected through surveys and 
student tests, although some research Is available that describes 
classroom processes In greater detail (e.g., Stake and Easlcy, 
1978). But there Is no common method nor even objective. Some 
surveys and tests use whole populations, others are based on 
national or state aamples, still others are characterized by self- 
selection of participants. Surveys may be cross-sectional, docu-- 
minting a single point In time, they may be repeated at Irregular 
Intervals, they may be designed as longitudinal studies or repeated 
annually. Obviously, periodic replication of studies Is necessary 
If temporal trends are to be Identified. Yet there are constraints: 
careful thought must be given to reducing the response burden In 
surveys, the dlsiniptlon that sometimes accompanies classroom obser- 
vation and other qualitative research, and the unforeseen negative 
consequences of Increased student testing. One way of limiting both 
the expense and the Interference of periodic surveys, testing, and 
qualitative research may be to set up a carefully selected panel of 
schools, with systematic rotation of schools Into and out of the 
panel, to provide a consistent data base. 

• Much of the data used In the reports critical of education come from 
national surveys or nationally administered tests. However, educa- 
tion In the United States Is decentralized and, despite some ten- 
dencies toward conformity, quite diverse In Inputs, processes, and 
outcomes. The richness and sometimes even the meaning of Informa* 
tlon Is obscured by reporting only national averages. Moreover, 
nationally aggregated statistics are of limited use In formulating 
policy at the state and local levels — and, after all. It Is the 
states and localities that largely determine what happens In schools 
In this country, not a national body. Therefore, If the condition 
of science and mathematics education Is to be portrayed so as to 
Inform key people and policy makers Involved In education, Indi- 
cators must be selected to be useful at the state and local levels 
as well as at the national level. 

• There Is one Important national goal not well met In science and 
mathematics education, the goal of equal opportunity. Information 
pertinent to this goal ought to be available. Therefore, It is 
Important to collect certain data by gender and minority status, 
such as different enrollment rates in mathematics and science 
courses. Other demographic descriptors may also be Important, for 
example, density of population or economic characteristics of dif- 
ferent communities. 



• Some Indicators may have to be represented by different measures at 
different levels of education. For example, the teaching of science 
and mathematics In elementary school Is not generally provided by 
specialist teachers and enrollment Is not recorded by specific 
courses > as Is the case In high school. A. special problem In this 
regard is the middle or Junior high school, which may b^ organized 
either like elementary or like secondary school. 

• Supposing that Indicators of mathematics and science education were 
available, how should they be Interpreted? Most commonly, indi- 
cators are used to make comparisons over time: Have test scores 
risen or fallen in the last year, the last decade? Are students 
taking fewer or more science courses than last year, five years ago? 
Comparisons can also be made among groups or geographic entitles* 
for example in examining distributional issues. Temporal trends 
can, of course, be established within comparisons of population 
groups, e.g., over the last five years, has mathematics course 
enrollment of females increased or decreased more rapidly (slovly) 
than that of males? A third basis for comparison is to e<xtablish 

an ideal value for an indicator and compare the recorded . .lue to 
the ideal, as is attempted in teacher demand-and-supply studies. 
The problem with this method is that establishing ideal values is 
usually difficult, but all three methods of interpreting indicator 
values are appropriate, given proper caution. 

Findings on the Information Pertinent 
to Selected Indicators 

The committee published its first report in April 1985 (Ralzen and 
Jones, 1985). Below is a summary of the findings the report on the 
adequacy of the currently available data and information. Recommenda- 
tions for Improvements are also given. 

The Quantity and Quality of Teachers 

Findings 

• Aggregate estimates of teacner supply and demand mask great dif- 
ferences among regions of the nation, states, and local school 
districts within states. 

• All estimates of teacher supply and demand are accompanied by large 
uncertainties. 
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with respect to supply, there are three major gapi In knowledge; 

(1) The data on the actual numbers of teachers assigned to m^^the- 
metlcs and science classes are Inadequate, especially as 
Aggregated at the national level* 

(2) The number of Inactive teachers who return each year to fill 
vacancies Is unknown* Since ^he cumber of trained teachers 
who do not enter teaching or who leave teaching Is sizable, 
this represents a considerable resource* The number cr 
teachers drawn from the Inactive pool may Increase as deslr^ 
able Job opportunities arise* 

(3) The moFt recent data on the annual supply of newly certified 
entrants to teaching are four years old* Hence, the effects 
of current Incentives to draw people Into the field ar . 
unknown* 

With respect to demand, there are four unknowns: 

(1) While enrollments are dropping, vacancies tend to be filled 
with teachers from other fields who have tenure In a district, 
rather than with new entrants certified In the field with 
vacancies* This practice, the extent of which Is unknown, 
reduces the demand for additional teachers, even though It may 
be det rimer tal to the quality of science and mathematics 
teaching* 

(2) The extent to which school systems will seek to replace out- 
of'*fleld teachers or will choose Instead to provide In-servlce 
training is unknown* Such choices will in part be Influenced 
by state and federal support policies for teacher education 
and In part by local board policies and teacher contracts* 

(3) To the degree that increased high school graduatica require^ 
ments will entail having to offer more courses in mathematics 
and science, teacher shortages will be aggravated, but hov 
much is unknown* 

(4) Demand forecasts are generally based on extrapolation of cur- 
rent conditions, taking account of likely changes in enroll- 
ment, class size, and curriculum* They do not taice into 
account possible structural changes in the education system* 

Since the committee completed its review, several studies have been 
Initiated to address some of these gaps* 



with respect to the quality of teachers, the following problems 
Kere Identified: 



• Adequate Information Is lacking on the qualifications of the 
teachers who are responsible for teaching mathematics and science 
In high school, middle/ Junior high school, or elementary school. 

• Information on certification, the only proxy available for qualifi- 
cation, Is lacking for all but new entrants, although data on a 
national sample of the teaching force are now being collected. 

• Even when available. Information on certification Is of question- 
able use as a measure of qrallflcatloa because state certification 
requirements and preservlce college curr5cula reflect a wide range 
of views on what constitutes a qualified or competent teacher in 
mathematics or science. Moreover, teachers currently certified 
obtained their certification at different times that may have 
required different types of preparation; therefore, certification 
even within the same state does not connote equivalent preparation. 

Conclusions and Recommendations 

• A suitable Indicator to assess the sufficiency of secondary school 
science and mathematics teachers would be either the ratio of or 
the difference between projected demand and anticipated supply of 
qualified teachers. The ratio would Indicate how close to balance 
demand and supply are; the difference would Indicate the number of 
teachers that need to be added or that exceed the demand. The con- 
struction of such an Indicator on teacher demand and supply Is at 
present not feasible at the national level because of the lack of a 
meaningful common measure of qualification. 

• Individual states and localities might construct this type of Indi- 
cator by using certification as an approximation for qualification 
or developing alternative criteria for teacher competence. In each 
case, an adequate determination would entail estimates of both 
demand and supply under alternative sets of assumptions about antic- 
ipated enrollments In mathematics and science classes and new 
entrants Into the teaching of these fields. Aggregation of the 
state data might provide a useful national picture, especially If, 
In addition. Information was reported concerning differences among 
states. 

• The disparate views on teacher qualification and the variation In 
certification standards Indicate the need to rethink the Initial 



preparation and continuing training appropriate for teachers with 
Instructional responsibilities In science and mathematlca. Guide- 
lines that have been prepared by professional societies need to be 
considered by the wider educational community, Including bodies 
responsible for the certification of teachers and accreditation of 
teacher education programs. Requirements should be detailed sepa- 
rately for teachers In elementary school (grades I to 5 or 6), 
middle or junior high school (grades 6 or 7 to 8 or 9), and high 
school (grades 9 or 10 to 12), with particular attention to require- 
ments that can be translated Into effective college curricula and 
In-servlce education for teachers. 

• The development of guidelines for the preparation and continuing 
education of teachers would be advanced If the attributes of suc- 
cessful teaching In science or mathematics were better understood. 
Further research Is necessary on the relationships between teacher 
training and student outcomes; for example, the effects on student 
achievement of different types of preservlce and In-servlce train- 
ing and of teaching experience. Current Initiatives to augment the 
pool of science and mathematics teachers should be monitored to 
assess their effectiveness. 

Curriculum Content 

Findings 

• Although commonly used textbooks and tests Introduce a modicum of 
similarity In the range of topics generally treated within a year's 
course of Instruction, emphasis varies from text to text, cl j to 
class, and test to test. Hence, for the nationally normed achieve- 
ment tests often used at the elementary and middle school levels, 
there may be a discrepancy between a student's opportunity to learn 
and the subject matter covered on the test, while at the same time 
the student may have learned considerably more than the test 
Indicates. 

• To a large extent, the content of Instruction Is based en the text- 
book used In a class, yet there Is no continuing mechanism to 
encourage periodic and systematic analysis of the use and content 
of science and mathematics texts. 

• At the secondary school level, and particularly In mathematics, 
course titles are a questionable Indicator of content studied. The 
current practice of accepting similar course titles as representing 
exposure to similar material Is likely to produce d'ta of question- 
able quality. 
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• Ihere are no established standards for content derived either from 
past practice, practice elsewhere, anticipated need, or from theo- 
retical constructs developed, say, from the nature of the discipline 
being taught or from learning theory. Until some consensus can be 
reached on Instructional content that represents desirable alterr.a- 
tlves for given learning goals, It Is premature to suggest a spe- 
cific Indicator for this area« 

• Information on what Is currently tau(;ht should be collected and 
aualyzed, and reviews of the curriculum should be done by scien- 
tists, mathematicians, and other experts in the dlsc'<plines as well 
as teachers and educators. The reviews should evaluate material 
covered at each grade level or by courses, such as first-year 
algebra, or Introductory biology; consider relationships among grade 
levels or courses; and identify the knowledge and skills expected 

of students at the completion of each grade or course. Such reviews 
are ni^eded in conjunction with addressing the critical matter of 
what content should be taught in mathematics and science* 

• At a minimum, periodic surveys should be conducted to determine the 
relative frequency of use of various mathematics and science text- 
books at each grade level in elementary school and for science and 
mathematics courses in secondary school. Timing of surveys should 
take into account the common cycles of textbook revision. 

• Surveys of textbook use should be followed by content analyses of 
the more commonly used texts. Analyses should proceed along peveral 
different lines: balance between the learning of recorded knowledge 
(concepts, facts) and its application (process), emphasis given to 
specific topics, adherence to the logic of a discipline, opportunity 
and guidance for student discovery of knowledge, Incorpoiatlon of 
learning theory. 

• Intensive studies should collect information from teachers and stu- 
dents on topics actually studied within a given grade or course. 
Observation of samples of individual classrooms can help to document 
the content of instruction. Such studies could help to Inform cur- 
riculum decisions by local districts, even though the results may 
not lend themselves to generalization over a state, let alone over 
the United States as a whole. 

• Improved definitions of secondary school courses, based on their 
content, should be developed. As a first step, use of a standard- 
ized course title list, such as the Classification of Secondary 
School Courses (Evaluation Technologies, Inc., 1982), should be 
considered. 
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• Critical analysis of standardl::ed tests should continue so as to 
establish their degree of correspondence to the Instructional con- 
tent of the class subjects for which they are used* Consideration 
should be given to Inviting the judgment of teachers v.and older 

^dents) concerning the students* opportunity to learn tho material 
chat is covered on each test* 

Instructional Time and Course Enrollment 

Findings 

• For elemental schools, not enough data are available to discern 
clear trends over the last 20 years with respect to amount of 
instructional time spent on mathematics and science* Existing 
loformatlon, however, points to great variability from class to 
class in the amount of time given to instruction in general and to 
each academic area specifically* 

• A number of problems attend enrollment data currently available: 
uncertainties generated by using self-reports, differences in ques- 
tions and method from survey to survey, and ambiguities created by 
similar course titles in mathematics that refer to different content 
or different levels of instruction* 

Conclusions aikd Recommendations 

• The average amount of time per week spent on mathematics -tristructJon 
and on science Instruictlon should be measured periodically for 
samples of elementary schools* This measure would serve as an 
indicator of length of exposure to pertinent subject matter; values 
can be compared for different years* Care must be taken, however, 
to ensure common understandings in collecting measures of time as 

to what constitutes science or mathematics instruction* Time given 
to mathematics or science, expressed as a percent of all instruct 
tlonal time, would indicate the priority given to these fields* 

• Efficiency of instruction shoulrl be assessed by comparing allocated 
time with instructional time and with time that is actually spent 
on learning tasks that appear to engage students, as established by 
observation* 

• Experimentation and research should be carried cut to develop a 
proxy measure for time spent on instruction that would permit col- 
lecting the pertinent information at reasonable costs* 
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• For grades 7 to 12 , enrollments In matheniatlcs and sclerce courses 
at each grade level and cumulatively for the six years of secondary 
school or fi>r the three or four years of senior high school shou?d 
be systematically collected and recorded. Alternatively, the mean 
number of years of mathematics or science taken or percentages of 
students taking one, two, or three or more years of such courses 
can be used as a measure. 

• The disparities in mathematics and science enrollment among various 
population groups warrant continued monitoring, so that distribu- 
tional inequj' ^es can be addressed. National data on student enrol- 
lments collected in connection with the periodic surveys recommended 
above may be insufficient for this purpose. States should consider 
biennial or triannual collection of enrollment data by gender, by 
ethnicity, and by density of the school population. 

Jtudent Outcomes 

Findings 

• It has proved difficult with current test methodology to construct 
t^sts that can be used for large numbers of students and yet are 
adequate for assessing an individual's cognitive processes, for 
example, the ability to generalize knowledge and apply it to a 
variety of unfamiliar problems. However, existing tests of mathe- 
matics and science of the kind employed by NAEP, HSB, and lEA are 
sufficiently valid for the purpose of indicating i cup achievement 
levels. 

Conclusions and Recommendations 

• Systematic cross -sectional assessments of general student achieve- 
ment in science and mathematics, such as the ones carried out 
through NAEP, should be carried out no less than every four years 
to allow comparisons over relatively short periods of time. The 
samples ou these assessments should continue to be sufficiently 
large to allow comparisons by ethnic group, gender, region of the 
country, and t >e of community (urban, suburban, rural, central 
city). 

• Longitudinal studies such as High School and Beyond are important 
for following the progress of students through school and later and 
should be maintained. 

• International assessments in mathematics and science education such 
as those sponsored by lEA need to be carried out at least every 10 
years. 
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• Developmental work on tests Is needed to ensure that they assess 
student leamlog considered useful and Important. Instrument i used 
for achievement testing should be reviewed from time to time by scl * 
entlflc and professional groups to ensure that tbey reflect contemp- 
orary knowledge deemed to be Important for students to learn. Such 
reviews may lead to periodic changes In test content — an objective 
that muf.t be reconciled with the goal of being able to compare 
student achievement over time. 

• Work Is needed on curriculum-referenced tests that can be used on a 
wider than local basis, especially for upper-level courses. This 
work will require careful research on the content of lastructlon, 
tests constructed with a common core of Items, and altsmative 
sections of tests to match currlcular alternatives. 

• Assessmenlis shOv'ld include an evaluation of the depth of a student *s 
understanding of concepts, the ability to address nonroutine prob- 
lems, and skills in the process of doin^ mathematics and science* 
Especially for science, it 5s desirable that a test involve some 
hands-on tasks. 

Current Work 

The committee's firnt report represented a preliminary selection of 
indicators and review of relevant data and infoniatlon. The report was 
criticized by scientists and mathemat.' ians for coacen Crating on conven- 
tional indicators and current data bases rather r:haa developing imagina- 
tive new indicators that would provide more penetiating insights on the 
condition of science and mathematics education. With support from the 
National Science Foundation, the committee is continuing its work. Some 
of the problems discussed ab( ve are receiving further attention, for 
example, defining teaching elfectiveness^ developing indicators of the 
quality of curriculum content, and improving assessment of student per- 
formance. Also, some potential indicators identified but not selected 
for discussion in the first report are being reexamined. 

lu the fall of 1985, the committee held a workshop involving some 
5C outside experts to develop improved approaches to indicators. Three 
aieas of schooling input and three areas of schooling outcomes were 
addressed: financial investment at the federal and state and local 
levels In mathematics and science education, teaching effectiveness, 
curriculum quality, learning of mathematics and science, motivation and 
attitudes, and scientific literacy. At this stage, it is not possible 
to report the committee's conclusions and recommendations, buc 1 will 
try to highlight progress to date. 
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FlLanclal Indlc^ttors 

AL the federal level, current Dudget Information does not allow a 
separate accounting for federal resources invested In science, mathe-* 
matlcs, and technology education. There are several reasons: with few 
exceptions, the budgets of agencies that support relevant activities do 
not Include separate line Items for science and mathematics education; 
progra&s designed as supporting science and mathematics education vary 
from agency to agency as to what subjects, educational levels , and popu- 
lations art Included; and agencies have different priorities regarding 
salient areas of policy Interest and therefore maintain different bud- 
geting categories* Unless considerably more resources were to be spent 
on creating compatible budgeting systems across agencies that would make 
possible identification of investments in mathematics and science edu- 
cation, a financial indicator at the federal level will be difficult to 
create. 

In any case, federal financial indicators will be useful only if 
they can be considered in conjunction with state and local financial 
data* Federal resources are only a small part of the total educational 
effort, e&pecially in precollege science and mathematics education* 
Hence, the federal contribution can best be appreciated in relation to 
the larger context, particularly if it could be shown how the federal 
funds are leveraging scate and local funds* However, at these levels 
as well, a major constraint is the current variability in recording 
financial and accounting data* Moreover, school budgets generally do 
not reflect Investments in instructional programs, and the budget cate- 
gories that are in common use do not make it possible to generate the 
requisite figures* 

Any attempt to profile financial information indicative of invest- 
mcats in science and mathematics education will require a significant 
R&D program* A usc^ful initiative would be to experiment with a pilot 
management information system on science and mathematics education in a 
few selected districts, perhaps some 20 sites* Such a project would 
entail adoption of a standard copt accounting rirrangement by the pilot 
districts aeveloping the needed reporting capacity at the local 
level to track funding of science and mathematics activities* Criteria 
for selecting pilot districts might include demographic characteristics 
(e*g*, urban, suburban, rural) and character of the mathematics/science 
programs (e*g*, programs meet minimum standards, programs are recognized 
as high-quality)* If the feasibility of such an approach can be demon- 
strated, further Issues that will need to be resolved include correc- 
tions for ccst-of -living and other market conditions that vary among 
localities, the need to modily the data base o/er time as schooling 
practices change, and changing inputs to education, such as the changes 
in the demand for and supply of teachers* 
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Assessing Teaching Effectiveness 

As pointed out .^.n the committee's first report, a key problem in 
creating an indicator on teacher supply Is defining teacher quality* 
Attributes that characterize effective teachers of mathematics and 
sciet.ce it elude Intellectual currloslty, subject matter knowledge « and 
ability to convey subject matter and Intellectual curiosity to students. 
All three attributes, not Just one or two, are Important to effective 
mathematics and science teaching. The reason for stressing the combined 
Importance of these attributes it? to forestall the adoption of policies 
raising the average level of Just one attribute — say, subject matter 
expertise — ^at the expense of lowering the level of the other two. 

Identifying attributes Important to effective science and mathe- 
natics teaching is only the first step; ther^ remains the thorny prob- 
lc»m of itieasuring these attributes. To assess Intellectual curiosity, 
one may wish to consider teacher? ' use of time both inside and outside 
the classroom. For example, infcrmation might be collected on the 
extent to which teachers use their own time to keep up with their sub- 
ject area, whether they take courses on their own, and whether they read 
such periodicals as Science and Scientific American . Bowever, there 
are many ways of displaying Intellectual curiosity, and it would be mis- 
leading to restrict measures to participation in a small, finite set of 
activities. Using professional time within school to pursue intellec- 
tual Interests requires that time is sex: aside in the normal school day 
for such purposes. Appropriate space that allows teachers to think, 
read, and talk about ideas with colleagues also is required. Measures 
of such preconditions for maintaining an active and productive concern 
with one*s teaching responsibilities might lead to desirable changes in 
the school environment provided for teachers. 

One would assume that knowledge of subject ^^t-ter, the sacond 
attribute, ought to be easiest to define and asse^^a. Most recent 
reports on improving the quality of teachers have stressed the need for 
Increased subject matter preparation. There is, however, a surprising 
lack of evidence in the literature on the relationship between increased 
knowledge of subject matter and teaching effectiveness. Obviously, 
teachers must understand the content they are expected to teach, but it 
is not clear what level of subject-matter knowledge beyond this ulnimtim 
will impro e their performance. They also need to have a sense of the 
structure of the discipline they teach and of the relevant importance 
of dltferent facts associated with the discipline. This is not neces- 
sarily engendered by taking more undergraduate courses. Nevertheless, 
there appears to be consensus that high-school erythema tics and science 
teachers, as a minimum, should possess the skills of an undergraduate 
major in the subject area. Thus, a gross general indicator would be to 
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collect Information on the proportion of teachers who have undergraduate 
majors la thn subjects they teach. Because undergraduate majors In a 
particular scl^biice or mathematics vary enormously In substance and dlf-- 
flculty, this gross Indicator should be augmented by accreditation 
models baaed on peer review, that Is, judgments by subject-matter 
experts on what constitutes a reasonable academic program. Since many 
high school science teachers teach more than one science, data also need 
to be collected on the percentage of students taught by teachers having 
adequate subject master preparation, as defined above. 

The Issue of adequate mathematics preparation for elementary-school 
teachers might be resolved by analyzing the mathematics content of the 
grades K-6 curriculum and then basing measurements of teacher quality 
on this analysis, for example, assessing the competence of teachers In 
solving problems that Involvp fractions and decimals* With respect to 
the science knowledge to be squired of elementary school teachers. It 
Is not possible at this tlm to define an Indispensable core, since the 
science curriculum Itself Is so variously and vaguely defined and — In 
fact — often absent altogether In any form recognizable as science by 
expert observers. 

Measures of teachers* competence In conveying subject-matter know- 
ledge and Intellectual curiosity to their students need to be based on 
minimum skills required of the teacher: exposing students to Important 
concepts, presenting multiple representations of concepts, trying to 
understand students* thinking, asking students to explain their answers, 
giving clear directions. Observational techniques could be used to 
determine the extent to which teachers of mathematics and science 
exhibit these behaviors. Development of reliable Indicators of such 
teacher behaviors are likely to be difficult and expensive, requiring 
extensive training of observers and observation of particular teachers 
for several hours, preferably on different days. Moreover, such Indi- 
cators probably will not distinguish Inspired teaching from merely ade- 
quate teaching, but they may Identify whether basic conditions are 
satisfied for Instilling In students both a knowledge of and positive 
attitudes about science and mathematics. 

Quality of the Curriculum Content 

Indicators are needed for all four aspects of curriculum content: 
the planned curriculum (e.g., state and local curriculum guidelines)* 
the Intended curriculum (e.g., the content of textbooks, laboratory 
exercises, and lesson plans), the content actually presented to the 
student, and the content learned by the student. If Indicators of these 
differently defined aspects of curriculum content were available, com- 
parisons could be made, for example, becween what state and local policy 
makers Intend and what goes on In the classroom. 
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Assessment of the content of Instruction In mathematics and science 
cacnot proceed without a framework that represent^^ the structure of the 
subject matter and desirable learning goals, or alternatives among 
goals. Therefore, such frameworks need to be developed for currlcular 
units. The frameworks must give attention both to the quantity of sub- 
ject matter covered and to the depth of coverage. They must also con- 
sider over what unit of time curriculum should be defined — a school 
term, a grade In school, or a longer period of time such as grades K-6. 
Possible currlcular units might be: K-8 mathematics, college-preparatory 
mathematics (algebra I, geometry, algebra II, analysis, calculus, and 
statistics), secondary-school mathematics for the non-coUege bound, 
K-8 science, college-bound scleace (biology, chemistry, physics), and 
secondary science for the non-college bound. The frameworks should meet 
some general criteria: they should array major processes, emphases, or 
principles In the curriculum against content topics rather than simply 
list detailed topics; they should represent the best thinking of a com- 
bination of disciplinary specialists and specialists In the design of 
curricula aad in teaching the subject; they should be conceived to lead 
practice, rather than representing a least common denominator of current 
practice; an<^ they should be flexible, presenting a commonly agreed-on 
core and allowing for major options or alternatives In the content pre- 
sented In states, localities, schools, and classrooms. Over time, these 
frameworks should be regularly and critically reviewed no as to reflect 
developments li the disciplines. 

Once a framework Is developed. It can serve as the basis for a 
series of analyses. One analysis would sample state and local curric- 
ulum objectives, guidelines, and testing programs to determine which of 
the elements In the framework are covered and to gauge the depth of 
cov^.age and the variability In coverage among school systems. A second 
stage would be to Identify the most frequertly used textbooks and to 
use the framework to determine which elements are covered. Since text- 
books represent the primary tool for planning curriculum content, this 
stage would yield Indicators of the planned curriculum. Analysis should 
be done often enough to account for periodic textbook revisions. 
Another us^ of the framework would be ae a basis for assessing the con- 
tent actually covered In classrooms. One way to do this would be to 
observe classes, but this Is expensive and time-consuming* Instead, a 
sample of teachers could be asked whether they covered various topics 
and to Indicate the depth of coveire^ge. Such surveys could be conducted 
In conjunction with student testing and would yield measures of stu- 
dents* "^opportunlty-to-leam** similar to those used In the lEA studies. 
Finally, the curriculum framework could be used to review the content 
of existing student achievement tests and design new ones. This would 
tie content analysis of the curriculum to Indicators of student achieve- 
ment. 
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Assessing Learning in Science and Mathematics 

The present system for assessing student progress in acquiring know- 
ledge, skills » and understanding in science and mathematics relies pri- 
marily on tests composed of multiple-choice items* ^uch items are used 
by NAEP in assessments at the national level and at state, district, 
and local levels. Multiple-choice tests are worth retaining because 
they provide an efficient and economical method for assessing the extent 
to which students have acquired necessary factual information and such 
elementary procedural skills as the algorithms involved in arithmetical 
computation* 

There are, however, some Important ways in which the current system 
for assessing student achievement is deficient: 

1* Oae should be able to assess those educational objectives that are 
poorly measured by present methods. Existing multiple-choice tests 
appear to exclude a variety of critically important educational 
goals • 

2* An improved system should provide diagnostic information that would 
be ubeful in helping students who fall to understand or master 
certain skills. 

3* An improved system should help teachers make wise decisions about 
what to teach and how to teach it. Available assessment ir ite rials 
appear to lead teachers to focus on the low-level objectives that 
are reflected in multiple-choice tests. Imprcved tests should 
provide models of performance reflecting the more complex cogni- 
tive skills, models that should be emulated by both teachers and 
students. 

4. An improved system should produce tests that are coachable only in 
the good sense that teaching a student to deal with the test prob- 
lems is equivalent to teaching the knowledge and skills the test 
is intended to measure rather than improving test scores without 
improving skills. 

5. An assessment system should be based on some understanding about 
curricular priorities. Unfortunately, the present fragmentation 
and^lack of consensus with regard to curriculum goals, especially 
in science, constitutes a potential barrier to the development of 
instruments for assessment* 

Some innovative assessment methods are under development. Contin- 
uing advances in cognitive theory promise to provide ideas regarding 
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what skills should be assessed, and computer science promises better 
tools with which to assess them. For example, research on the miscon- 
ceptions that many students have regarding physical phenomena provides 
Instances of how better understanding of cognitive processes may lead 
to more useful assessments. Another example Is the research on differ- 
ences between novice and expert problem solvers* Such studies strongly 
suggest that methods for discovering how one represents a problem 
Internally and how these representations are altered with training and 
practice would be of great value In Instruction. A third example Is 
research concerned with the extent to which certain aspects of problem 
solving can be performed automatically, with a minimum of attention, 
through pattern recognition and other short-cuts* 

Computer simulations of "hands-on** performance In doing experiments 
allow assessment of student skills In scientific thinking* It Is also 
possible, through analyzing protocols of student Input, to develop a 
refined picture of a student's processes for solving problems* Coach- 
ing systems developed for computer use could be adapted ^or assessing 
performance In very complex domains* Thus, the compute opens up sev- 
eral new possibilities for assessment* Paper-and-pencll simulations 
represent another option: Descriptions of proposals, research Inves- 
tigations, and experimental results can be presented to students for 
analyses, criticism, and explanation* Trained coders can then assess 
the quality of student responses. 

The methods of assessment described above obviously cannot compete 
with multiple-choice tests from the standpoint of economy and effici- 
ency. However, the cost of using the methods should be justifiable, 
not only because they would provide Information for a far more accurate 
and complete assessment of Instruction, but also because they would be 
directly useful In the educational process. Exercises derived from the 
methods could be used for practice and to provide Information for remed- 
iation, and assessments based on the methods should raise educational 
standards by providing models of performance to le emulated by both 
students and teachers. 

Developing alternatives to multiple-choice tests will require con- 
siderable creative effort and funding Investment. Serious consideration 
should be given to the creation of a National Library of Science and 
Mathematics Assessment Materials. This library would be a compendium 
of bclence and mathematics exercises designed both for testing and for 
classroom use as Instructional aids. The materials In the library would 
be publicly available, and a system for receiving feedback from users 
could facilitate Improvements In content and methodology. The library 
materials should be created by a consolidated effort of scientists. 
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educators, and testing professionals. Its philosophy should be to teach 
science and mathematics not as rote knowledge but as active problem- 
solving and to emphasize basic concepts and methodologies. 

Assessment of Attitudes and Motivation 

Current research has not been highly success^cul In establishing 
causal relationships between student attitudes and motivation and stu- 
dent achievement, although frequent correlations have been noted. The 
apparent lack of connectedness could be a function of three things: 

1. The constructs of attitudes and achievement that have been used 
are unrelated. 

2. The measures of attitudes /motivation are faulty. 

3. The measures of achievement are faulty. 

Generally, It has been assumed that the problems reside In either 1 or 
2, but It Is quite possible achievement measures In current use are not 
sufficiently broad to connect up vlth attltudlnal Indices. In any case, 
motivation and attitudes are worth assessing as educational ends In 
themselves, regardless of achievement. 

The choice of Indicators In this domain should be guided by con- 
structs that can explain attitudes of students towards science and 
mathematics and their motivation for learning and performing In these 
fields. Three constructs considered Important are engagement — choosing 
to attend to Ideas and applications of science and mathematics; compe- 
tence — ^knowing what cicrategles are needed to be successful In these 
fields and being In command of the strategies; and autonomy — the sense 
that one chooses to engage In an activity for one's own purpose. 

Several principles should govern the development of Indicators of 
attitudes and motivation toward science and mathematics. First, quali- 
tative as well as quantitative methods will be necessary to assess the 
Identified constructs. National and state level assessments should be 
done through large-scnle surveys, but at the same time smaller sub- 
samples should be Interviewed Intensively so as to connect results from 
quantitative and qualitative research. Second, for both qualitative 
and quantitative studies, measures need to be domain-specific as well 
as assessing more general processes. And third, assessment strategies 
must be developmentally appropriate. The meaning of science will differ 
between 3rd grade and 12th grade, so the methods for measuring both 
attitudes and achievement must be sensitive to the student's stage of 
comprehension. 
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Regarding competence and autonomy, survey questionnaires and inter- 
views should establish how students perceive their control over under- 
standing mathematics and science in and out of school, how they perceive 
their capacities » and the degree to which children experience choice in 
learning matUematics and science. Questions should also probe the 
extent to which children view mathematics and science as being connected 
to other meaningful aspects of their lives. More indirect indicators 
may have to be developed to assess engagement. One such indicator might 
be the extent to which students choose to participate in opportunities 
to learn science; another might be the career interests of students. 
Various non-school behaviors also may reflect student engagement with 
science — a predisposition to read science materials and view science 
television and film programs, voluntary participation in science pro- 
jects, using deductive (scientific) learning to evaluate information 
outside of science class, and s Iving puzzles, word problems, and 
computer simulations for fun. 

Indi':ators of General Science Literacy 

Any useful attempt to determine the status of scientific literacy 
in ttie population must reflect an informed view of what constitutes 
scientific literacy. At least four dimensions need to be addressed: 

1. The nature of the scientific world view; 

2. The nature of the scientific enterprise; 

3* The role of science in human affairs; and 

4. Scientific habits of mind. 
Each of these dimensions is elaborated below. 

The scientific world view is made up of ideas and beliefs at vari- 
ous levels of complexity. Grand conceptual schemes are formulated to 
bring together and reduce to order large numbers of ideas, theories, 
and observations that are of lesser generality. Particular discoveries 
and ideas may lead to the development of new theories and are powertul 
tools for making sense of natural phenomena. The various sciences tend 
to operate in the context of firmly held notions about the natural world 
which is assumed to be understandable and not capricious. 

laymen also need to understand that science is a social activity 
carried out by scores of individuals who collaborate over time and 
place. It is important to be aware that scientific findings are always 
tentative, open to modification and correction as new discoveries are 
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made. The scientific enterprlp-* Is conducted by Individuals who sub- 
scribe to a set of value commitments In principle, but In as much as 
they are human beings, theire are variations In the actual behavior of 
Individual scientists* 

To a very large extent, peop. .^ encounter science as It affects human 
events. The application of science to such matters as health, agricul- 
ture, and the environment engage not only scientists but also a large 
number of policy makers and Individual citizens. In a scientifically 
literate society, people will need to understand some of these rela- 
tionships. For example, scientists engaged in public matters may behave 
differently than when they are ac<:lng as reseachers. Tensions will 
always exist between science and society because science is able to deal 
with uncertainty and ambiguity to a degree that is not possible by 
organizations that must make policy decisions or by individuals who must 
make personal ones. AJ.so, the differences and overlaps between science 
and technology need to be understood as well as the fact that both are 
likely to have unpredicted Impact on society. 

Scientific habits of mind include familiarity with natural phenom- 
ena; identifying questions or formulating hypotheses rcdatlve to prob- 
lems posed by these phenomena; identifying and seeking out relevant 
information; using that information to test the hypotheses, answer the 
questions, or creating new hypotheses; and offering arguments and coun- 
terarguments that can be tested by reference to data or accepted 
principles. 

Any plan to generate indicators of scientific literacy should try 
to estimate the degree to which a population possesses the kind of know^ 
ledge and Intellectual skills outlined in the above four categories. 
In doing this the following attributes ought to prevail. 

1. A single measure will not do for these multiple dimensions of a 
complex B€t of characteristics. The indicators should be matched to the 
model of scientific literacy. 

2. Indicators should be flexible, changing over tliiie in form and 
content to mfitch changes in the model Itself. 

3. £ven so, there must be enough continuity in the indicators to 
maKe it possible to track populations over time* 

4. llie indicators should recognize that ther^ is no absolute level 
of literacy and that various levels of attainment in different compon- 
ents of a community or population group are likely. 
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5. Any measures used to generate indicators should be supplemented 
by research. 



6* Indicators may be expressed in terms of descriptive patterns of 
behavior and other non-numerical ways. 

7. Indicators can be expressed as group literacy as well as indi- 
vidual literacy* Any conclusions based on the indicators must relate 
only to the unit of analysis. 

Current techniques of conducting polls. Interviews, and case studies 
should all be considered in generating indicators* Traditional methods 
may work reasonably well to assess knowledge, but they should be further 
developed to probe also the population's understanding of the nature of 
science and its role in society* It is particularly important and dif- 
ficult to obtain reliable estimates of problem-solving skills* Assess- 
ment of such skills need to go beyond individual paper-and-penctl tests 
and should include observation and analysis of individual and group 
responses to carefully selected phenomena involving real objects and 
filmed sequences of events* 



At this time, the committee is collecting reactions from various 
groups to its suggestions* I look forward to the discussion period to 
elicit yours* 
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